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Abstract 

Let X be a complex Banach space of dimension at least 2, and let 5 be a mul- 
tiplicative semigroup of operators on X such that the rank of ST — TS is at 
most 1 for all {5*, T} C S. We prove that S has a non-trivial invariant subspace 
provided it is not commutative. As a consequence we show that S is triangular- 
izable if it consists of polynomially compact operators. This generalizes results 
from 0] and [H. 
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1. Introduction 

Throughout the paper, let A be a complex Banach space of dimension at 
least 2. A subspace of X means a closed linear manifold of X. Trivial subspaces 
of X are {0} and A. The dual space of A is denoted by A*. By an operator on 
A we mean a bounded linear transformation from A into itself. By / we denote 
the identity operator. The Banach algebra of all operators on A is denoted by 
B{X). We denote by T* the adjoint operator of T e B{X). The notation [S, T] 
is used as an abbreviation for the commutator ST — TS, where S,T G B{X). 
Given y G X and (j) G the rank-one operator y eg) </> on A is defined by 
{y®(j))x = (j){x)y. 

A subspace of A is invariant under an operator T £ B{X) whenever 
T{M.) C M. Let C be a collection of operators in B{X). A subspace of A 
is invariant under C if is invariant under every T G C. If, in addition, the 
subspace Ai is invariant under every S G B{X) that commutes with all operators 
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of C, Ad is said to be hyperinvariant under C. A collection C is triangularizable 
if there is a chain of invariant subspaces for C which is maximal as a subspace 
chain. 

In many situations the existence of a non-trivial invariant subspace already 
implies triangularizability, as the Triangularization Lemma shows (see [1] or 
0). In order to recall it, some definitions are needed. Let C be a collection of 
operators in B{X). If M and M are invariant subspaces under C with Af C M, 
then C induces a collection C of quotients as follows: for T C, the operator 
f e C is defined on M/Af by 

f (a; + TV) = Tx + Af . 

Any such C is called a sei 0/ quotients of the collection C. A property of collec- 
tions of operators is said to be inherited by quotients if every set of quotients of 
a collection having the property also has the same property. 

Lemma 1.1. (The Triangularization Lemma) Let V be a property of collec- 
tions of operators that is inherited by quotients. Lf every collection of operators 
(on a space of dimension greater than one) which satisfies V has a non-trivial 
invariant subspace, then every collection satisfying V is triangularizable. 

A collection of operators is called an (operator) semigroup if it is closed 
under multiplication. It is well-known fact that every commutative semigroup 
of matrices is triangularizable. In 1978 Laffey extended it as follows (see \4(). 

Theorem 1.2. If S is a semigroup of matrices such that the rank of [S,T] is 
at most 1 for all {S*, T} C S, then S is triangularizable. 

As a generalization of the preceding theorem, Radjavi and Rosenthal proved 
the following theorem (see Corollary 2]). 

Theorem 1.3. If S is a semigroup contained in the Schatten class Cp such that 
the rank of [S,T] is at most 1 for all {S,T} C S, then S is triangularizable. 

Applying the remarkable result of Turovskii 0] the last theorem can be easily 
extended to compact operators on X (see 0, Theorem 9.2.10]). 

Another infinite-dimensional extension of Laffey's result was shown by the 
group of authors in ^1]. Recall that an operator T on X is said to be algebraic 
if there exists a nonzero complex polynomial p such that p{T) — 0. 

Theorem 1.4. Let S be a semigroup of algebraic operators on X such that the 
rank of [S,T] is at most 1 for all {S*, T} C S. Then S is triangularizable. 

In 01 we showed the following theorem related to the subject. 

Theorem 1.5. Let S be a non- commutative semigroup in B{X) generated by 
two elements. If the rank of [S,T] is at most 1 for all {S,T} C S, then S has 
a non-trivial hyperinvariant subspace. 
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The preceding theorem is an easy consequence of the following theorem 
Theorem 2.2]. 



Theorem 1.6. Let A and B be operators on X such that the rank of [A, B] is 
1 and the ranks of each of [A^^B], [A,B'^], [A'^,B'^], [AB,BA] are at most 1. 
Then {A, B} has a non-trivial hyperinvariant suhspace. 

It is clear from the proof of Theorem 11.61 that the following is also true. 

Theorem 1.7. Under the assumptions of Theorem \l.b\ let [A^B] — yf^xj) for a 
non-zero vector y G X and a non-zero functional (j) € X* . Then 4>{y) — 0, and 
so [A,B]^ =0. 

The proof of Theorem 11.61 is essentially based upon the following (easily 
proved) observation Lemma 2.1]. 

Lemma 1.8. Let y X and (j) e X* he non-zero vectors. Assume that z £ X 
and -0 g X* are such that the rank of the sum y ® (j) A- z ® ip is at most 1. Then 
either z is a multiple of y or ip is a multiple of (p. 

2. Results 

The key result of this paper is the following theorem. 

Theorem 2.1. Let S be a semigroup in B{X) such that the rank of [S,T] is at 
most 1 for all {S*, T} C S. Suppose that [A, B] — y ® 4> for some {A, _B} C S 
and for some non-zero vector y £ X and non-zero functional (p G X* . Then 
cj){Cy) =0 for allC eSU{I}. 

Proof. With no loss of generality we can assume that €S — S, that is, S 
is closed for scalar multiples of its members. Since </)(?/) = by Theorem 11.71 
we must prove that (j>{Cy) = for all C £ 5. Assume on the contrary that 
(j>{Cy) ^ for some C £ 5. As €S — S, we can assume that <t>{Cy) = 1. Since 
(j)(y) = and (C*(/>)(y) = 4){Cy) = 1, neither y is an eigenvector of C nor (j) is 
an eigenvector of C* . Therefore, it follows from Lemma fTTSl that the rank of 

[AB, C] - [BA, C] = [[A, B], C] = (y ® - C{y ® (j)) ^ y ® {C*<j)) - (Cy) ^ 

is equal to 2. Since the ranks of [AB,C] and [BA, C] are at most 1, they are 
both equal to 1. In fact, we must have 

[AB, C] = {ay + (3Cy) (7^ + 5C*(f>) 

for some scalars a, (3, 7 and 6. We claim that S ^ 0. Assume on the contrary 
that S — 0. Clearly, we can assume that 7 = 1. An application of Theorem 11.71 
for the pair {AB, C} yields (/'(ay + (3Cy) = 0, and so /3 = 0. Then the rank of 

[BA, C] = [AB, C] - [[A, B],C]^y®iacb- C*dp) + {Cy) ® 
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is equal to 2. This contradiction proves the claim. Now, with no loss of generality 
we can assume that (5=1, and so 

[AB, C] = (ay + pCy) {jcp + C*(j)) and 

[BA,C] = [AB,C]-[[A,B],C] = y®(a7(/.+ (a-l)C*0)+Cy®((l+/37)<^+/3C*0). 

Since rank [BA, C] < 1 and Cy is not a multiple of y, by Lemma [LSI there exists 
a scalar A such that 

ajcj) +{a- 1)C*0 = A((l + + (3C*<P). 

Since C*0 and (j) are linearly independent, we have 

A(l + Pj j) = a-f and A/3 = a - 1. 

Eliminating a we obtain that A = 7, and so a = 1 + (3^. An application of 
Theorem O for the pair {AB,C] yields (7^ + C*4)){ay + pCy) = 0, which 
implies that 

1 + 2/37 + mc^v) = 0. 

It follows that /3 7^ 0, and we may define fc = + 7. Therefore, we have 

[AB, C] = /3{ky + Cy) ® (70 + C*cj)), 
[BA, C] = /3(7y + Cy) ® (fc0 + C*^) 

and 

^(C^y) = -A; - 7. 

Now, the rank of 

[AB, C^] = [AB, C]C + C[AB, C] = 

= I3{ky + Cy) ® (7C*0 + (C*)^^) + (i{kCy + C^y) {jcj) + C*dp) 

is at most 1, and so, by Lemma ll.81 again, one of the following two cases must 
occur. 

Case (I): kCy + C^y — X{ky + Cy) for some scalar A. Applying the functional 
(j) to this equation we obtain that A = fc + (f>{C'^y). Since (f>{C'^y) = — fc — 7, we 
have A = —7, so that 

C^y + ik + "f)Cy + kjy = or {C + k){C + j)y = 0. 

Now, decompose X as X ^ lin{y} © lin{C?;} © ker(</i) n ker(C*0). Note that 
ker((/)) = lin{?/} © ker(0) n keT{C*4>). With respect to this decomposition the 
operator C has the matrix 

/ -/C7 ci3 \ 

C = [ 1 -fc - 7 C23 . 

V C33 / 
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li X e ker((/)) n kei{C*4>), then (j){Cx) = 0, so that Cx 6 ker((^) = linjy} © 
ker(0)nker(C*0). This implies that C23 = 0. Since [v4, B]x = for aU x e ker((/)) 
and [A, B]Cy — y, we have 

/ 1 

[A, B] = 
\ 

Since 

[AB,C]y^l3{ky + Cy) 

and 

[AB,C]Cy^-(3kiky + Cy), 

we have 

k 

[AS, q = /5 I 1 -/fc I /? I 1 I ( 1 -/c ) . (1) 








Similarly, we obtain that 




^21 


d22 - dii 


d23 





-d2i 








-dsi 






7 

[BA, C] = /? I 1 -7 I = /? I 1 I ( 1 -7 ) 



Denoting _D = AB = {dij)^j^i, we have 
[AB, BA] ^ [A, B]D - D[A, B] - 



Since rank [Ai?, < 1, we conclude that d2i — 0, and either ^23 = or 
c?3i = 0. Therefore, we must consider two subcases. 

Subcase (la): ^23 = 0. Then 

[AB,C] = [AB^dnI,C] = 

di2 - cisdai -(fc + j)di2 + ^7(^22 - dii) - £13^32 ^13233 - 213(^33 - ^ii) 
^22 — dii —di2 —di3 

d'i2 - 033^31 -kjdsi - (fc + 7)d32 - 033^32 d3lCl3 + ^33233 " 233(^33 

Comparing with ([T]) and simplifying we get 

di3 = 0, di2 = (3k, d22 = dii + /3, 

Cisdai = 0, 013^32 = 0, ^32 = 033^31. (2) 

We thus have 



dii 


13k 









dii+P 




and 


^31 


d32 


^33 / 
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BA = AB- [A, B]= \ dii+13 

^31 C^32 1^33 

Now compute the commutator 

[ABBA, C] = [AB, C]BA + AB[BA, C] = 

1 -fc ) 




dii 


/?7 





dii + 


^31 


d32 






(i 


)■■ 








Pk 

dn+l3 I /3 ( 1 1(1 -7 ) 

cJ32 ^33 

/ dii7 + /3/c 

-Mii-l 0)+/? dn+f3 1(1 -7 0) 

\ (i3i7 + d32 

Since its rank is at most 1, two subsubcases are possible by Lemma 11.81 

Subsuhcase (lal ): (dii7 + fik, dn + (3, ^317 + ^32) — i-L{k, 1, 0) for some scalar fi. 
Eliminating fi we obtain that dn = and 

^317 + ^^32 = 0. (3) 

We now compute the commutator 

[BAAB, C] = [BA, C]AB + BA[AB, C] = 

Pk 

= /3 I 1 I ( 1 -7 ) I (3 

c'31 0^32 ^33 
-fc 




f 1 








)< 


1 - 











Pi 








p 





^31 


dz2 


^33 


(0 


1 


0)- 




(1 -k ) . 

Since its rank is at most 1, we conclude that c?3ifc + c?32 — 0. Using ([3]) we 
obtain that ^31 — and o?32 = as fc 7^ 7. Therefore, the two-dimensional 
subspace lin{y, Cy} is invariant under {C,AB,BA}. Let C, D and E denote 
the restrictions to this subspace of C, I? = AB and E = BA, respectively. Then 



C 



-fc7 

1 — fc — 7 
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I) = /3|^^j(0 1) and E ^ (3 i^^^ j { I ). 

Now, we could apply Theorem ll.2l to obtain triangularizability of the semigroup 
generated by C, D and E which would contradict the fact that the commutator 



[D -E,C] = 



1 —k ~ 7 
-1 



is not nilpotent. However, there exists a quick way to get a contradiction di- 
rectly. Since 

i)C = /5 ^ ^ ( 1 -fc - 7 ) and i^C = /? ^ ( 1 ~k~j ), 
we have 

[DC,E]^l3'l^(^'iyO -A:)+(^|)(l fc + 7)| and 

[^C,^]=/3^|(|)(0 -7)+(^ fc + 7) 

Since the ranks of both commutators are at most 1, we conclude first that fc = 
and then 7 = 0. This is not possible, and so the subsubcase (lal) is finished. 

Subsubcase (Ia2): (dii, —kdu — 1,0) = fi{l, —7, 0) for some scalar n. It follows 
that dii = 11 = —f3. In this case we compute 

[CAB, BA] = C[AB, BA] - [BA, C]AB = 

10)- 

~f3 (3k 

dai d^2 c'as 

( -/? pk ) , 

where we have used two equalities from ([2]) . Since the rank of this commutator 
is at most 1, we obtain that 7 = and (^32 = 0. Similarly, we obtain that 

[CBA, AB] = C[BA, AB] - [AB, C]BA = 

(010)- 





7 





-/3 











^31 


d33 


k \ 








; 





) . 

It follows that fc = 0, so that 7 = fc = 0. This contradiction completes the proof 
in this subsubcase, and so Subcase (la) as well. 

Subcase (lb): dsi = 0. Then 

[AB, C] = 



dii 


di2 


di3 







d23 





d32 


d33 




( dn 


di2 


)i 





d22 




V 


d32 




-kj ci3 

1 -/c-7 
C33 

Comparing with ([1]) we obtain di2 = (3k, ^22 — dn +/3 and c?32 = 0. Therefore, 
the two-dimensional subspace lm{y,Cy} is invariant under {C,AB,BA}, and 
so we can proceed as in Subsubcase (lal) to get a contradiction. Denote by C, 
D and E the restrictions to this invariant subspace of C, Z? = AB and E = BA, 
respectively. Then 



C = 



-kj 

1 ~k ~ 7 



By Theorem 11.21 the semigroup generated by {C, Z?, E\ is triangularizable. 
However, this implies a contradiction, as the commutator 

{E-EM=[^ \ "^17^ 

is not nilpotent. This concludes this subcase, and so Case (I) as well. 

Case (II): ^C*(j) + {C*)'^(j) = A(7(/) + C*(j)) for some scalar A. Computing both 
sides of this equality at the vector y yields A = 7 + (j){C^y). Since 4>{C^y) = 
—k — 7, we obtain that A = — fc. Therefore, we have 

+ (fc + 7)C*0 + kjcj) = 0. 

Thus, the semigroup S* = {S* : S E S} satisfies the same conditions as the 
semigroup S in Case (I). Indeed, if x 1-^ i^^, denotes the isometric embedding of 
X to X**, then we have [B*,A*] ^ (j) ® Fy, Fy{(l)) = (l){y) = and Fy{C*(t)) = 
{C*(j)){y) — 4>{Cy) ^ 0. So, we can use Case (I) to obtain a contradiction. This 
completes the proof of the theorem. 
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Corollary 2.2. // S is a non- commutative semigroup in B{X) such that the 
rank of [S, T] is at most 1 for all {S, T} C S, then S has a non-trivial invariant 
subspace. 

Proof. Since S is not commutative, we have [A, B] ~ y(E)<j> for some {A, B} C S 
and for some non-zero vector y £ X and non-zero functional (f) £ X*. Then the 
closed linear span of the set {Sy : S € SU {/}} is a non-zero subspace invariant 
under S. Since 4>{Sy) = for all S" S 5 U {/} by Theorem l2.H it is contained in 
ker(^), and so it is non-trivial. 

The preceding corollary does not hold for commutative semigroups, since S 
may be generated by an operator without non-trivial invariant subspaces. In 
order to cover the commutative case as well, the following property of operators 
was introduced in 2]. Let TZ be the property of operators on Banach spaces of 
dimension at least 2 such that: 

(a) TZ is inherited by quotients, 

(b) each commutative semigroup of operators with the property TZ has a 
non- trivial invariant subspace. 

Since every non-zero compact operator on an infinite-dimensional Banach 
space has a non-trivial hyperinvariant subspace by the famous Lomonosov's 
result , the property of being a compact operator is an example of such prop- 
erty TZ. Another example is the property of being an algebraic operator, as 
eigenspaces of an algebraic operator are hyperinvariant subspaces. 

The main result of the paper is the following theorem. 

Theorem 2.3. Let S be a semigroup of operators on X with the property TZ. 
If the rank of [S, T] is at most 1 for all {S, T} C S, then S is triangular izable. 

Proof. By the Triangularization Lemma, it suffices to show that S has a non- 
trivial invariant subspace. This is true by Corollarv l2.2l if S is not commutative. 
Otherwise, this holds by the condition (b) of property TZ. 

Compact and algebraic operators are special cases of polynomially compact 
operators. Recall that an operator T on X is said to be polynomially compact 
if there exists a nonzero complex polynomial p such that the operator p{T) is 
compact. Triangularizability of collections of polynomially compact operators 
was studied by Konvalinka in 3|. Since the Lomonosov's result is strong enough 
to give non-trivial hyperinvariant subspaces of polynomially compact operators 
(that are not algebraic), the property of being a polynomially compact operator 
is also an example of property TZ. So, we have the following corollary of Theorem 
12.31 that extends both Theorems 11.31 and 11.41 

Corollary 2.4. Let S be a semigroup of polynomially compact operators on X . 
If the rank of [5, T] is at most 1 for all {5, T} C S, then S is triangularizable. 
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